dunning-kruger effect
Do Code Models Suffer from the Dunning-Kruger Effect?
Singh, Mukul, Chatterjee, Somya, Radhakrishna, Arjun, Gulwani, Sumit
As artificial intelligence systems increasingly collaborate with humans in creative and technical domains, questions arise about the cognitive boundaries and biases that shape our shared agency. This paper investigates the Dunning-Kruger Effect (DKE), the tendency for those with limited competence to overestimate their abilities in state-of-the-art LLMs in coding tasks. By analyzing model confidence and performance across a diverse set of programming languages, we reveal that AI models mirror human patterns of overconfidence, especially in unfamiliar or low-resource domains. Our experiments demonstrate that less competent models and those operating in rare programming languages exhibit stronger DKE-like bias, suggesting that the strength of the bias is proportionate to the competence of the models.
- North America > United States > Florida > Miami-Dade County > Miami (0.04)
- Europe > Ireland (0.04)
- Asia > Singapore (0.04)
- Asia > Indonesia > Bali (0.04)
- Overview (1.00)
- Research Report > New Finding (0.94)
Humans and AI: Organizational Change
According to McKinsey, "Research shows that 70 percent of complex, large-scale change programs don't reach their stated goals. Common pitfalls include a lack of employee engagement, inadequate management support, poor or nonexistent cross-functional collaboration, and a lack of accountability." Last year I was doing some spring cleaning and looking for space in my home office for a digital piano. As I pulled books from my bookcase, packing them into boxes to go into storage, I found my Blockbuster Video membership card. I'd tucked it inside a book as a bookmark.
- Media > Television (0.31)
- Leisure & Entertainment (0.31)
- Information Technology (0.31)
Council Post: Pairing AI With Human Judgment Is Key To Avoiding 'Mount Stupid'
Malinka Waliyadde is Co-Founder & CEO of Alpha Health, the first Unified Automation company for revenue cycle management in healthcare. It's an irony of the human condition that often, when we are at the depth of our ignorance, we are at the height of our confidence. Technology, machines and AI (artificial intelligence) don't automatically fix this folly. Like humans, these systems do not always understand when they are wrong about their own competence (for example, if the state of the world changes from when the AI model was originally trained). But integrating people into a system that uses machine learning goes a long way toward eliminating blind spots.
You Aren't So Smart: Cognitive Biases are Making Sure of It
According to Wikipedia, cognitive biases "are tendencies to think in certain ways that can lead to systematic deviations from a standard of rationality or good judgment, and are often studied in psychology and behavioral economics." Far more than simply an exercise in academia, cognitive biases have all sorts of practical impacts on our lives, whether or not we admit it. A very broad umbrella, cognitive bias comes in many forms, as evidenced by the fact that Wikipedia lists over 170 of them. Some of these biases are more prevalent in certain areas of life than in others. Below is an infographic from Business Insider, of all places, which is an elementary summary of what it refers to as "20 cognitive biases that screw up your decisions." But how do these cognitive biases relate to real life?